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Abstract 

Examining repeated classroom en¬ 
counters over time provides a compre¬ 
hensive picture of activities. Studies of 
instructional practices in classrooms 
have traditionally relied on two meth¬ 
ods: classroom observations, which are 
expensive, and surveys, which are lim¬ 
ited in scope and accuracy. Teacher logs 
provide a “real-time” method for col¬ 
lecting data on classroom practices by 
giving teachers a tool to document and 
reflect about specific lessons and the 
impact they may have had on their stu¬ 
dents. Logs can collect data for review 
by teachers, their colleagues, administra¬ 
tors, and researchers. These self-reported 
data, collected online repeatedly over 
a specified period of time, present a 
series of snapshots that capture ongo¬ 
ing classroom practices and lesson strate¬ 
gies. In this article, we describe the use 
of teacher logs to understand classroom 
practices by highlighting the kinds of 
activities teachers emphasized and their 
perceptions of effectiveness. In a Nation¬ 
al Science Foundation-funded evalua¬ 
tion of a high school reform program, 
we used online teacher logs to under¬ 
stand the education delivery in Science, 
Technology, Engineering, and Mathe¬ 
matics (STEM) subjects in redesigned 
North Carolina high schools. We find 
that teacher logs are an appropriate, 
efficient, and useful tool for document¬ 
ing practice. Teachers can use log data 
to reflect on practice, determine areas 
of strength and challenge, and set goals 
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for personal improvement. Administra¬ 
tors can use logs to support teachers in 
examining practice and setting goals. 

Introduction 

The study of instructional practices 
in classrooms has traditionally relied on 
two methods: classroom observations, 
which are expensive, and surveys, which 
are limited in scope and accuracy. Teach¬ 
er logs provide a “real-time” method for 
collecting data on classroom practices 
by giving teachers a tool to document 
and reflect about different lessons and 
the impact they may have had on their 
students. Logs can be developed to col¬ 
lect data for review by teachers, their 
colleagues, administrators, and research¬ 
ers. These self-reported data, collected 
online repeatedly over a specified period 
of time, present a series of snapshots that 
capture ongoing classroom practices and 
lesson strategies. This type of data col¬ 
lection can help teachers reflect on their 
implementation of new practices and 
programs at school. Here, we focus on 
teachers in schools that have undergone 
Science, Technology, Engineering, and 
Mathematics (STEM) reform. 

Many practitioners and evaluators 
agree that dynamic, interactive instruc¬ 
tional practices are a key component 
of Science, Technology, Engineering, 
and Mathematics (STEM) education. 
Approaches such as creativity strategies, 
problem-based learning, and learning 
through design are particularly effective 
for reinforcing STEM-based material 
(Clark & Ernst, 2007). If instruction mo¬ 
tivates students, then students are likely 
to value their educational endeavors and 
perhaps even seek similar educational 


experiences in the future (Durik & 
Harackiewicz, 2007). Ultimately, the in¬ 
struction students receive should inspire 
and motivate them to pursue STEM ca¬ 
reers. To connect content to students’ in¬ 
terests, STEM teachers are encouraged 
to embed content in real-life practical 
problems (Community for Advancing 
Discovery Research in Education, 2012). 
As the National Science and Technology 
Council’s Committee on STEM Educa¬ 
tion points out, the President’s Council 
of Advisors on Science and Technology 
(PCAST) concluded that STEM teachers 
should have “enough content knowledge 
to link STEM to compelling real-world 
issues, model the process of scientific 
investigation, effectively address student 
misconceptions, and help their students 
learn to reason and solve problems like 
mathematicians, scientists and engineers” 
(Committee on STEM Education, 2013, 

p. 18). 

North Carolina New Schools (NCNS) 
along with the New Tech Network (NTN) 
supported the development of STEM 
high schools with guidance and profes¬ 
sional development. The authors worked 
with NCNS to study a set of schools 
that had received support from NCNS. 
NCNS’s vision for STEM emphasized 
making connections in the fields of math 
and science; meaningfully integrating 
technology; and helping students cul¬ 
tivate creativity and develop problem 
solving, communication, and collabo¬ 
ration skills. NCNS provided profes¬ 
sional development to teachers to help 
them create classrooms with these char¬ 
acteristics. Some schools working with 
NCNS also received professional devel¬ 
opment from NTN. This National Science 
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Foundation-funded study examines how 
much these professional development ex¬ 
periences influence the daily classroom 
practices of the participating teachers. We 
developed online teacher logs to obtain 
evidence as to whether teachers routinely 
used instructional practices aligned with 
the NCNS and NTN goals and profes¬ 
sional development. 

Teacher Logs 

When Ball, Cambum, Correnti, Phelps, 
and Wallace (1999) conducted a pilot 
study of the usefulness of teacher logs as 
a web-based tool, they asked teachers to 
make note of a number of variables that fell 
into four main categories: (a) the nature of 
student work, (b) the specific activities in 
which students engaged, (c) the teacher’s 
actions, and (d) the topics and content 
covered. The authors found that by docu¬ 
menting classroom practices, such as how 
students were grouped, what materials they 
used, which types of activities were used 
to engage students, what the teachers were 
doing, and how the content was presented, 
they could form general insights as to the 
typical practices employed in the class¬ 
room. In studying the use of logs to de¬ 
termine the quality of instruction. Rowan 
and colleagues (Rowan, Jacob, & Correnti, 
2009; Rowan & Correnti, 2009) identified 
four dimensions of teacher-student inter¬ 
play that were important to assessing the 
quality of an educational setting. Dimen¬ 
sions they studied included: 

• Social relationships among stu¬ 
dents and between students and 
teacher, 

• Coherence and cognitive demand 
of content, 

• Pedagogical practices of the 
teacher, and 

• Order and organization of the 
classroom. 

The original research with online logs 
(Ball, Camburn, Correnti, Phelps, & 
Wallace, 1999; Rowan & Correnti, 2009; 
Rowan, Jacob, & Correnti, 2009) was 
validated in elementary school math¬ 
ematics and reading classes. These stud¬ 
ies provided the framework in which RTI 
developed teacher logs to study selected 
STEM high schools in North Carolina. 
Ball’s work was particularly applicable 


to RTFs study as it focused on obtaining 
a more complete picture of what teach¬ 
ers do when teaching math (e.g., what 
materials they use, how they approach 
instruction) and analyzed the impact of 
interventions on instruction. 

RTI’s study examined the characteris¬ 
tics of classroom instruction in the STEM 
schools and the extent to which it reflect¬ 
ed NCNS goals. We wanted to determine 
the priorities teachers set for instruction, 
the variety of activities in which stu¬ 
dents were engaged, the way students 
used technology in the classroom, and 
how frequently lessons incorporated vari¬ 
ous attributes. Teacher logs provided a 
vehicle for continuously monitoring the 
pedagogical techniques teachers utilized 
to engage students in problem solving and 
collaborative practices. 

Method 

Site-Based Implementation 

Because multiple site visits would 
have been cost-prohibitive and possibly 
disruptive to the school day, we designed 
teacher logs based on research (cited 
above) which indicated that these logs 
are an effective way to collect reliable 
data on the frequency with which teach¬ 
ers use different types of instructional 
activities when multiple observations are 
not an option. Teachers were asked to log 
about one specific class period (e.g., geo¬ 
metry first period, earth science fourth 
period) throughout the study in order 
to provide an ongoing series of snap¬ 
shots of that class and the experience 
of its students. To minimize the burden 
on teachers, the log had only five open- 
ended questions, three of which were only 
asked on the first administration. The 
remaining questions required teachers to 
rate the lesson attributes on scales from 1 
(low) to 5 (high). Specific descriptors on 
the scales varied based on the questions 
asked (e.g., “never” to “often,” “not im¬ 
portant” to “moderately important”) but 
always indicated that 1 was the lowest 
rating and 5 was the highest rating. 

We designed four sections similar to 
the dimensions recommended in the ear¬ 
lier research: (a) general information and 
classroom context (teacher and student 


demographics, teacher preparation and 
licensure, classroom materials available), 
(b) lesson description (content and topics, 
student tasks), (c) instruction (materials 
and student engagement strategies), and 
(d) implementation (teacher reflection on 
lesson success/progress). Pilot testing of 
the log instrument showed that test sub¬ 
jects took between 10 and 12 minutes to 
complete a log. Because we would not 
expect general information and classroom 
context to change much over the course of 
the term, questions pertaining to general 
information and classroom context were 
asked only on the first log. Sections on the 
lesson description, instruction, and imple¬ 
mentation were included in every log be¬ 
cause they could vary within each lesson. 

We used the logs in two academic years. 
Each year, we selected four teachers from 
each school to complete a predetermined 
number of logs. The typical configura¬ 
tion included two math and two science 
teachers; but, when possible, we included 
health, agriculture, and engineering teach¬ 
ers. These options were limited in the small 
schools participating in this study. Because 
the research suggested that the usefulness 
of the log data tapered off somewhere 
between 10 and 20 logs (Rowan and 
Correnti, 2009), we set the threshold at 14 
in the first year and 12 in the second year. 

In the first year, we collected logs 
from 10 schools, and in the second year, 
we limited data collection activities to 
three schools. The Year 1 response rate 
varied by school, and some teachers in 
some schools did not participate. In Year 
2, RTI collected logs from 12 teachers 
at three schools. We also added face-to- 
face training and reduced the logging 
period to one semester, which increased 
response rates. For one third of the logs, 
we set target dates for completion to co¬ 
incide with our site visits, and teachers 
chose when to complete the remainder. 
In year 2, we requested a total of 144 
logs and received 122. 1 


1 In school 1, all four teachers completed the 
required 12 logs. In school 2, one teacher 
quit logging after one entry and another 
teacher logged eight times. In school 3, 
one teacher logged five times, while the 
rest completed the required 12 logs. 
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This article focuses on Year 2 because 
it had more complete response rates. 
The schools included in this year tend 
to serve students who were underrepre¬ 
sented in college and in STEM fields. 
Table 1 shows the characteristics of 
these schools. 

Two of the schools are in rural areas, 
and one is in a town. By design, these 
small schools serve fewer than 400 stu¬ 
dents. In all of them, at least half of the 
students are members of ethnic groups 
underrepresented in college and in STEM 
fields, and at least half are eligible for 
free- or reduced-price lunch. One of the 
schools (school 2) has faced challenges 
attracting and retaining highly qualified 
teachers, and 67 % of its teachers were 
novices (i.e., had fewer than four years 
of experience). Only about two thirds of 
school 2’s teachers were fully licensed. 
The turnover rate for school 2 was 50 %, 
while turnover rates for schools 1 and 2 
were 7% and 14%, respectively. In com¬ 
parison, state-level data show that 20% 
of all North Carolina teachers were nov¬ 
ices, 93% were fully licensed, and the 
teacher turnover rate was 16% (North 
Carolina Department of Public Instruc¬ 
tion, 2014). 

Technical Implementation 

RTI provided teachers with a training 
manual, a link to the online survey site, 
and a unique login identifier. We sent 
weekly emails throughout the study to 
remind teachers to log and record the 
activities for a specific lesson. Teach¬ 
ers could contact a designated RTI staff 
member if they had questions about the 
log. They were asked to avoid report¬ 
ing on a class that experienced atypical 


interruptions or shortened periods due 
to external influences such as school as¬ 
semblies, fire drills, or special programs. 
By compressing the 12 required logs into 
an eight-week period, teachers had less 
chance to forget and were able to get 
into the habit of completing logs. Each 
school subsequently received a case 
study report, with log data aggregated to 
the school in order to provide a picture of 
instructional practices across the chosen 
classes, rather than a picture of individu¬ 
al teacher practices. 

Because teacher logs are repeated sur¬ 
veys, we used an internally developed 
web-based survey tool for teacher log¬ 
ging. The survey tool required “emp¬ 
tying” after each use, so the data were 
transferred to a spreadsheet where they 
were collected and maintained through¬ 
out the course of the logging. Teachers 
could choose to complete the logs in a 
paper and pencil format. They received 
printed copies, which they could mail or 
fax. Teachers almost always chose the 
web-based version. 

Results 

A review of the log data collected 
from the 12 teachers who participated in 
the year 2 data collection revealed sev¬ 
eral interesting findings. For ease of re¬ 
porting, the 12 participating teachers are 
designated as Teachers A-D (school 1), 
Teachers E-H (school 2), and Teachers 
I-L (school 3). 2 


2 Teacher E logged once. Because all teach¬ 

ers completed the first section only once, 
her responses to questions in this section 
are included in analyses, but other portions 
of the single log she completed are not. 


Teacher Priorities in Lesson Planning 

In the first section of the logs, teachers 
responded to a list of lesson attributes, 
rating the importance of each attribute on 
a scale of 1 (low) to 5 (high) in develop¬ 
ing lessons and student activities (Table 
2). We compiled the list from national 
documents (National Research Council, 
2012; National Council of Teacher of 
Mathematics, 2014) that recommend 
classroom practices designed to engage 
and interest students. This question was 
intended to reveal the teachers’ beliefs 
and pedagogical practices and was asked 
only once, during the initial logging 
session. Teachers rated the importance 
of each element in a set of attributes, 
which included “ensure active partici¬ 
pation by all,” “encourage students to 
generate ideas,” and “make real-world 
connections.” 

Low teacher turnover, years of expe¬ 
rience, and shared professional devel¬ 
opment experiences seem to influence 
consistency of beliefs across STEM 
teachers within a school. Table 2 shows 
each teacher’s responses to questions 
about priorities in lesson planning, along 
with years of experience and subject 
taught. Overall, school 1 had a low turn¬ 
over rate and a relatively low percentage 
of novice teachers. School 1 was in its 
eighth year of operation, and all four 
STEM teachers joined the faculty when 
the school opened. Table 1 shows that 25 
percent of teachers at school 1 are novice 
teachers, so the STEM teachers are a lit¬ 
tle more experienced than the school av¬ 
erage. Three had more than 10 years of 
teaching experience. Teachers in school 
1 showed more consistency in rating 
the importance of lesson elements than 
teachers in other schools in the study. All 
teachers had participated in the specific 
professional development 3 provided to 
the school, and collectively, they indicat¬ 
ed that one of the greatest influences on 
their pedagogical approaches and teach¬ 
ing practice was the shared professional 
development they received from NTN 
or NCNS. Perhaps these years of shared 


3 Results about teacher professional de¬ 
velopment are available from the au¬ 
thors. 


Table 1: Characteristics of Schools in Academic Year 2012-13 


School 

Urbanicity 

Total 

students 

Percent under¬ 
represented 
minority 

Percent 
eligible for 
free or reduced- 
price lunch 

Percent 

fully-licensed 

teachers 

Percent 

novice 

teachers 

Turnover 

rate 

1 

Town 

128 

55.5 

61.7 

100 

25 

7 

2 

Rural 

217 

76.0 

57.1 

67 

67 

50 

3 

Rural 

148 

61.5 

63.5 

100 

50 

14 


Sources: National Center for Education Statistics (2014). Common Core of Data Public School Universe, 
2012-2013. Washington, D:C: U.S. Department of Education. 

North Carolina Department of Public Instruction (2014). North Carolina School Report Cards: 2012-13 
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teaching experiences influenced the con¬ 
sistency of their pedagogical beliefs. 

In contrast, in school 2, which had 
the highest turnover rate and highest 
percentage of novice teachers overall, 
three STEM teachers had fewer than 
three years of experience. One teacher 
had 32 years of experience. According 
to NCNS, “since the founding principal 
retired in at the end of the 2010-2011 
school year, the school has experienced 
tremendous and constant turnover” (per¬ 
sonal communication, NCNS staff, April 
1, 2014). Three of the STEM staff had 
not received the same level of profes¬ 
sional development. Teacher E was the 
only STEM teacher who attended the 
NCNS Summer Institute in June 2013, 
so the current STEM faculty had not 
shared a professional development ex¬ 
perience. Teacher E completed only one 
log, so we cannot determine her class¬ 
room practices, but Table 2 shows that 
the three-day institute did not prompt 
her to see importance in a number of 
the desired lesson attributes. Teacher H 
rated all 13 elements a 5, while the three 
newer teachers rated the elements with 
2s, 3s, and 4s, giving only seven 5s 
collectively. Of the 13 attributes, only 
“respect students’ contributions” and 
“embed problem solving” were rated 4 


or 5 (high importance) by all of the 
teachers in this school. 

In school 3, two of the teachers had 
fewer than six years of teaching experi¬ 
ence, while the other two were veterans 
with more than 10 years of experience. 
This school did not continue to receive 
direct NCNS services in 2013-14, and 
only teacher I attended the NCNS Sum¬ 
mer Institute in June 2013. Teacher J re¬ 
ported that support in previous years had 
guided the development of her instruc¬ 
tional practice. The current principal (in 
her second year) was a former science 
teacher at the school and had been there 
since its inception. In school 3, teachers 
ranked about half of the elements as im¬ 
portant (with a score of 4 or 5). 

Interestingly, the three math teachers 
with the least teaching experience (E, G, 
and J) gave most of the middle-level and 
low-importance ratings. They gave 20 of 
the 25 “3s” as well as the only “2s.” In 
neither school 2 nor school 3 did all of 
the teachers have a shared professional 
development experience with the NCNS 
or the NTN. 

Student activities during the lesson. 

Teachers responded to questions in the 
remaining sections of the logs each time 
they completed a log. The next section 
focused on describing that day’s lesson, 


and a key question asked teachers what 
students did during the day’s lesson. 
Pedagogical options included items involv¬ 
ing various levels of student engagement. 
Less-challenging activities included: lis¬ 
ten to a presentation by the teacher, per¬ 
form tasks requiring ideas or methods 
already introduced to the student, and 
assess a problem and choose a method to 
use from those already introduced to the 
student. Choices involving more critical 
thinking and problem solving included: 
perform tasks requiring ideas or methods 
not already introduced to the student, 
assess a problem and devise a creative 
solution, explain an answer or a solu¬ 
tion method for a particular problem, 
and prove that a solution is/isn’t valid 
or that a method works/doesn’t work 
for all similar cases. 

Logs indicated that teachers 
employed a variety of cognitive and 
pedagogical options for student engage¬ 
ment over time. Table 3 illustrates the 
range of cognitive processes—the means 
by which one acquires knowledge and 
develops understanding—recorded by 
the 11 teachers who completed multiple 
logs. Many teachers used all of the cog¬ 
nitive approaches listed over the course 
of the logging period. However, students 
in teacher B’s class engaged in very 


Table 2: Importance Ratings (1 -5) of Lesson Attributes by Teachers (5 = Very Important) 


Teacher -> 


School 1 



School 2 



School 3 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

Years teaching -A 

8 

18 

22 

15 

1 

0 

2 

32 

16 

5 

1 

17 

Subject*-^ 

S 

S 

M 

M 

M 

E 

M 

S 

M 

M 

E 

S 

Lesson attributes 













Ensure active participation by all 

5 

5 

4 

4 

3 

5 

3 

5 

5 

5 

5 

5 

Respect students’ contributions/opinions 

5 

5 

5 

5 

4 

5 

5 

5 

5 

5 

5 

5 

Embed opportunities for discourse 

5 

5 

4 

4 

2 

3 

3 

5 

4 

5 

4 

4 

Encourage students to generate ideas. 

5 

5 

5 

5 

3 

5 

3 

5 

4 

4 

5 

4 

Include challenging concepts 

5 

5 

5 

5 

3 

5 

2 

5 

4 

4 

5 

3 

Encourage collaboration 

5 

5 

5 

5 

4 

4 

3 

5 

5 

5 

5 

5 

Make real-world connections 

5 

5 

4 

5 

3 

4 

3 

5 

4 

3 

4 

4 

Develop scaffolded questions 

5 

5 

4 

5 

2 

5 

3 

5 

5 

4 

5 

3 

Allow for revisions 

5 

4 

4 

5 

3 

4 

3 

5 

4 

3 

5 

3 

Focus on “big” ideas 

5 

5 

4 

5 

3 

5 

2 

5 

3 

3 

4 

4 

Develop students’ confidence 

4 

5 

4 

5 

3 

4 

4 

5 

4 

4 

4 

4 

Excite my students 

4 

5 

4 

4 

3 

4 

3 

5 

4 

3 

4 

4 

Embed problem solving 

5 

5 

5 

4 

4 

4 

4 

5 

5 

4 

5 

3 


*Note: S=science, M=mathematics, E=engineering 
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few of these activities over 12 class pe¬ 
riods. Having conducted site visits, we 
know from teacher B that, during this 
period, students were working on a spe¬ 
cific project, and these listed activities 
may not have fit the project work. Spe¬ 
cifically, her students were constructing 
and decorating a booth for a health fair, 
where they would display what they had 
learned about nutritional benefits of vari¬ 
ous diets. Much of the work during the 
logging period was focused on building 
the booth and preparing the information 
packets for visitors, so opportunities for 
cognitive engagement were low. 

The activities in the table are listed 
in ascending level of rigor. It is strik¬ 
ing that some teachers rarely reported 
that students were expected to “listen 
to a presentation by a teacher.” Per¬ 
haps teachers believed that their own 
presentations were a less socially desir¬ 
able response to this item and chose to 
log about days that did not focus on 
their presentations. Or, perhaps the term 
“presentation” seemed more formal than 
leading a discussion about the course 
concepts or delivering a short lecture. 
In any case, we will use caution in inter¬ 
preting these results. None of these pat¬ 
terns would have been detected with 
only a single observation or one-time 
survey. 

A second key question asked how the 
students would go about the assigned task; 
in other words, what pedagogical strategies 
were being employed. So, while Table 3 
shows what students were expected 
to do (types of cognitive engagement). 


Table 4 illustrates how they went about 
the assigned tasks (variety of pedagogical 
options). Although teachers’ responses 
varied widely, the first two approaches, 
“answer recall questions orally” and 
“work on textbook, worksheet, or board 
work exercises,” together garnered a high 
number of responses although Teacher B 
did not report doing them at all. “Work 
on problems that have multiple answers 
or solution methods” and “engage in 
meaningful discourse of ideas, problems, 
solutions, or methods in pairs or small 
groups” also showed high response rates. 
Note that few teachers reported the “use 
of manipulatives”—visual and tactile 
tools—or “write extended explanations 
about science activities.” Only two teach¬ 
ers ever chose the “none of the above” op¬ 
tion, indicating that this list of activities is 
comprehensive. Teacher B, who selected 
this response in 25 percent of the lessons, 
seemed to be using different practices 
from other teachers. 

Use of Technology 

The next section of the logs focused 
on instruction and contained a question 
about the use of technology in the logged 
lessons. The question was intended to go 
beyond a “yes-no” inquiry about whether 
technology was used at all in a lesson 
and sought to determine the purposes 
for which the technology was used in 
each class. Choices included low-level 
activities such as “used for calculation” 
to higher forms of student engagement 
such as “manipulating variables in mod¬ 
els and simulations.” 


Technology remains an under¬ 
utilized tool in supporting lesson de¬ 
velopment, even in these redesigned 
STEM schools. Teachers in each school 
reported that students used technology for 
calculation purposes more than twice as 
often as they used it to manipulate vari¬ 
ables in models. The first three items, 
“to develop conceptual understanding,” 
“to learn or practice a skill,” and “for cal¬ 
culation purposes,” require only lower- 
level thinking skills with fairly passive 
student participation , while items such 
as “manipulate variables” and “collect 
data,” are more student-led and require 
critical thinking and active decision¬ 
making. Table 5 shows a significant 
drop in the use of technology for more 
student-led activities. 

Few teachers reported using mod¬ 
els and simulations as problem-solving 
tools. Models are animations of phenom¬ 
ena that can be manipulated repeatedly 
by varying the input variables. The re¬ 
sult of the manipulation simulates what 
would happen in real life and contributes 
to conceptual learning. The math and sci¬ 
ence concepts that can be studied using 
models and simulations extend the use of 
technology in the classroom far beyond 
that of a high-end calculator or online 
worksheet. Few teachers reported using 
technology to collect data - even though 
laptops, calculators, and cell phones can 
now collect data ranging from heart rate 
to humidity, which students could have 
analyzed. 

As with the use of the word “presen¬ 
tation” discussed earlier, there is some 


Table 3: Percent of Logged Lessons (n=122) in Which Teachers Reported Use of Various Cognitive Engagement Activities 




School 1 



School 2 



School 3 


Teacher -> 

A 

B 

C 

D 

F 

G 

H 

1 

J 

K 

L 

Cognitive Engagement Activities 

Listen to a presentation by the teacher 

58 

0 

42 

58 

33 

100 

8 

25 

33 

58 

40 

Perform tasks requiring ideas or methods already introduced 

92 

0 

58 

67 

83 

75 

92 

92 

75 

50 

100 

Assess a problem and choose a method to use from those 

67 

0 

42 

42 

67 

75 

33 

58 

75 

0 

0 

already introduced 

Perform tasks requiring ideas or methods not already introduced 

33 

25 

50 

50 

33 

50 

67 

16 

42 

50 

20 

Assess a problem and devise a creative solution 

42 

8 

8 

33 

67 

63 

25 

16 

33 

16 

0 

Explain an answer or a solution method for a particular problem 

83 

25 

42 

50 

50 

88 

58 

67 

92 

50 

20 

Prove that a solution is/isn’t valid or that a method works/ 

58 

0 

0 

16 

16 

38 

25 

0 

44 

0 

20 

doesn’t work for similar cases 


Note: Teacher G logged 8 lessons and Teacher L logged 5 lessons. Responses from Teacher E, who logged only 1 lesson, are not included in this analysis. 
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Table 4: Percent of Logged Lessons (n=122) in Which Teachers Reported Use of Various Pedagogical Strategies 




School 1 



School 2 



School 3 


Teacher-> 

A 

B 

C 

D 

F 

G 

H 

1 

J 

K 

L 

Pedagogical Strategies 

Answer recall questions orally 

58 

0 

33 

8 

92 

63 

75 

75 

50 

8 

0 

Work on textbook, worksheet, or board work exercises 
for practice or review 

58 

0 

67 

75 

33 

63 

92 

50 

92 

16 

20 

Work on problem(s) that have multiple answers or solution methods, 
or involve multiple steps 

67 

0 

50 

33 

67 

100 

42 

75 

92 

25 

40 

Engage in meaningful discourse of ideas, problems, solutions, 
or methods in pairs or small groups 

67 

16 

58 

67 

92 

75 

100 

75 

67 

83 

100 

Use manipulatives, games, or technology activities to 
improve recall or skill 

67 

8 

0 

0 

25 

25 

58 

0 

33 

8 

20 

Use manipulatives, games, or computer activities to explore concepts 50 
via models and simulations 

25 

0 

0 

16 

0 

58 

25 

16 

16 

0 

Write extended explanations about science activities—such as 
experimental design, data collection, and findings—or about 
mathematical ideas, solutions, and methods 

50 

25 

0 

25 

33 

38 

16 

16 

0 

0 

0 

Work on an investigation, problem, or project over an 
extended period of time 

42 

50 

33 

42 

50 

25 

0 

8 

33 

58 

20 

None of the above 

0 

25 

0 

16 

0 

0 

0 

0 

0 

0 

0 


Note: Teacher G logged 8 lessons and Teacher L logged 5 lessons. Responses from Teacher E, who logged only 1 lesson, are not included in this analysis. 


concern about the teachers’ interpreta¬ 
tion of the term “develop conceptual 
understanding.” This was intended as a 
higher-level use of technology, which was 
not witnessed in the classroom observa¬ 
tions conducted in this study. Perhaps 
teachers thought that using technology 
to learn or practice a skill was how one 
develops conceptual understanding. 

Teachers’ assessments of lessons. 
The final section of the log asked teach¬ 
ers to reflect upon the progress made in 
that day’s lesson. As noted above, the 
first time teachers completed a log, they 
responded to a question about the im¬ 
portance of various lesson attributes. In 
every subsequent log, teachers indicated 
the extent to which each attribute was a 
part of that day’s lesson on a scale of 1 to 
5. At no time did RTI expect that all attri¬ 
butes would be evident in all lessons, but 
rather that over time different attributes 
would play varying roles in the lessons 
as a whole. A look back at Table 2 shows 
that in the first log, most teachers rated 
most of the attributes as highly impor¬ 
tant. By repeating the question in each 
subsequent log, RTI hoped to compare 
the pedagogical beliefs expressed in the 
initial log to the actual classroom prac¬ 
tice (Table 6), noting the match between 


the intended curriculum and the enacted 
curriculum. 

Teachers’ assessments of their 
taught lessons indicate that they did 
not simply give a desired response - 
they acknowledged the difference be¬ 
tween the ideal (beliefs in Table 2) and 
the real (occurrence in Table 6). With 
repeated logs, we found differences in 
the way teachers responded to the inclu¬ 
sion of these attributes in their lessons. 
While many desirable lesson attributes 
earned 4s and 5s in Table 2, teachers 
were more selective in their responses 
and ratings when reflecting on their les¬ 
sons (Table 6). As logging continued, 
teachers seemed to respond thoughtfully 
to this question, and Is, 2s, 3s, and 4s be¬ 
gan to appear in the data, while the num¬ 
ber of 5s seemed to drop. However, we 
note that teacher H reported including al¬ 
most every attribute in every lesson, and 
that teacher G did not incorporate any 
of them in most of her logged lessons. 
Results from teachers listing almost all 
5 s or Os suggest that these teachers may 
have rushed through the log. 

Another way to study the data is to 
look at the occurrence, across a teacher’s 
12 lessons, of each lesson attribute listed 
in Table 2. By looking at how often each 


attribute was perceived by the teachers 
to be evident in the 12 logged lessons, 
we get a sense of which ones the stu¬ 
dents experienced more often. Whereas 
Table 6 shows the number of 5s teach¬ 
ers awarded themselves in each lesson, 
Table 7 shows the number of lessons in 
which each attribute was rated a 5. Look¬ 
ing at the data this way supports the ear¬ 
lier suggestion that teachers G and H are 
outliers whose data do not reveal much 
about their classroom practice. 

Table 7 presents the same lesson at¬ 
tributes as Table 2, but depicts them ar¬ 
ranged by frequency of occurrence. This 
allows an analyst to look for a variety of 
trends. For example, Table 7 shows that 
some of the attributes rated as occur¬ 
ring most often in all logged lessons are 
those with greater behavioral relevance, 
but less academic relevance. Those les¬ 
son attributes rated highest and most 
often included respect for student ideas, 
collaboration, and opportunity for dis¬ 
course. Academic attributes that were 
rated lower and occurred less frequently 
included making real-world connections 
and embedding problem-solving activi¬ 
ties. On their first logs, teachers did not 
rank either behavioral or academic attri¬ 
butes as more important. Repeated logs 
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Table 5: Percent of Logged Lessons (n=122) in Which Technology Was Used to Achieve a Specific Purpose 




School 1 



School 2 



School 3 


Teacher-> 

A 

B 

C 

D 

F 

G 

H 

1 

J 

K 

L 

Purpose of Using Technology 

To develop conceptual understanding 

92 

25 

42 

25 

50 

63 

83 

33 

25 

58 

40 

To learn or practice a skill 

92 

8 

58 

33 

16 

83 

75 

25 

33 

42 

0 

For calculation purposes 

67 

16 

50 

58 

8 

100 

33 

83 

0 

58 

0 

To conduct research (e.g. Internet) 

67 

42 

33 

8 

75 

0 

16 

0 

0 

8 

0 

To manipulate variables in models and simulations 

67 

0 

0 

16 

0 

13 

42 

25 

8 

8 

0 

to study outcomes 

As a presentation tool 

16 

8 

8 

8 

16 

0 

0 

0 

58 

16 

40 

For word processing or as a communications tool 

33 

58 

16 

0 

8 

13 

8 

0 

0 

0 

20 

(e.g., e-mail, Internet, Web) 

As an analytic tool (e.g., spreadsheets or data analysis) 

33 

16 

0 

8 

0 

0 

0 

0 

8 

8 

0 

To collect data (e.g., probeware, applications, sensors) 

16 

0 

0 

8 

8 

0 

16 

8 

0 

0 

0 


Note: Teacher G logged 8 lessons and Teacher L logged 5 lessons. Responses from Teacher E, who logged only 1 lesson, are not included in this analysis. 


show that in practice, they focused more 
on behavioral attributes. 

Discussion 

RTI used teacher logs to examine 
whether instructional practices in STEM 
classes were aligned with the goals the 
professional development teachers re¬ 
ceived through the NCNS or NTN mod¬ 
els. We found that teachers did employ 
a variety of cognitive and pedagogical 
options for student engagement over 
time, but that technology remains an 
under-utilized tool in supporting lesson 


development. Most teachers were able 
to link their many classroom activities to 
the program’s goals. 

We are encouraged by the useful role 
that logs played in gathering data on 
classroom instructional practices and stu¬ 
dent engagement. In this study, the logs 
revealed, through self-reported data, that 
the professional development provided 
by NCNS was not always implement¬ 
ed consistently across the classrooms. 
These differences seem to be related to 
teacher turnover and lack of institution¬ 
alized professional development. Low 


teacher turnover, years of experience, 
and shared professional development ex¬ 
periences may influence consistency of 
beliefs across teachers within a school. 

The variety of data gathered via the 
logs suggests that they provide a fuller 
picture of instructional activity than one 
gets from infrequent observations and 
end-of-year surveys. Unlike end-of-the- 
year surveys, that capture recollections 
of general practices over the course of a 
year, logs target specific practices on a 
specific days in a specific class. This data 
collection tool gave us the ability to de¬ 
termine the importance of specific lesson 
attributes, the frequency and variety of 
different student engagement strategies 
used by teachers, the purposes of tech¬ 
nology integration in lessons, and the 
match between what attributes teachers 
value in lessons and what they are actu¬ 
ally able to embed in their lessons. 

By repeatedly collecting this data over 
the term, we saw variation in teacher 
practices and observed that one teacher’s 
reported activities differed from most of 
her colleagues. We were able to see the 
variety of ways in which students used 
technology in their classes. A one-time 
survey could have had teachers rank 
classroom activities or the use of tech¬ 
nology with options like “never,” “rare¬ 
ly,” “sometimes,” “often,” and “always.” 
This approach would not have yielded as 
much specific information about instruc¬ 
tional practices. 


Table 6: Number of Attributes (n=13) Rated “5” by Teacher by Lesson (5=Attribute Achieved) 


Teacher A 


School 1 



School 2 



School 3 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

Lesson Number 













Lesson 1 

13 

8 

9 

9 

0 

7 

3 

13 

0 

2 

7 

9 

Lesson 2 

13 

9 

10 

6 


6 

0 

13 

8 

7 

5 

3 

Lesson 3 

12 

3 

6 

8 


5 

0 

13 

5 

0 

11 

4 

Lesson 4 

7 

10 

8 

13 


7 

0 

13 

10 

8 

6 

4 

Lesson 5 

4 

5 

10 

9 


8 

0 

12 

4 

0 

13 

6 

Lesson 6 

2 

11 

4 

2 


11 

0 

13 

8 

0 

7 


Lesson 7 

6 

12 

4 

5 


12 

0 

13 

3 

4 

5 


Lesson 8 

7 

8 

5 

4 


11 

0 

12 

3 

5 

3 


Lesson 9 

7 

10 

1 

7 


11 


13 

12 

1 

1 


Lesson 10 

4 

4 

0 

5 


4 


13 

6 

9 

6 


Lesson 11 

10 

3 

0 

6 


0 


13 

7 

9 

9 


Lesson 12 

5 

4 

1 

4 


9 


11 

2 

5 

7 



Note: Teacher G logged 8 lessons and Teacher L logged 5 lessons. Responses from Teacher E, who 
logged only 1 lesson, are not included in this analysis. 
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Table 7: Number of Lessons in Which Teachers Indicated that They Included Specific Attributes (maximum =12) 


Teacher A 

A 

B 

C 

D 

F 

G* 

H 

1 

J 

K 

L* 

Total 

School A 


School 1 



School 2 



School 3 



Attributes 













Respect students’ contributions/opinions 

7 

10 

8 

10 

11 

0 

12 

10 

8 

10 

1 

87 

Embed opportunities for discourse 

7 

10 

6 

10 

5 

0 

12 

10 

7 

10 

2 

79 

Encourage collaboration 

5 

8 

7 

10 

10 

1 

12 

7 

5 

6 

5 

76 

Encourage students to generate ideas 

10 

11 

3 

4 

11 

0 

10 

8 

5 

8 

1 

71 

Include challenging concepts 

5 

8 

3 

2 

7 

0 

12 

8 

6 

10 

3 

64 

Make real-world connections 

7 

11 

4 

3 

5 

1 

12 

4 

2 

8 

2 

59 

Ensure active participation by all 

7 

2 

6 

7 

7 

0 

12 

2 

8 

6 

1 

58 

Develop scaffolded questions 

9 

6 

3 

3 

6 

2 

12 

3 

3 

3 

2 

52 

Allow for revisions 

6 

3 

7 

11 

6 

0 

9 

4 

4 

1 

1 

52 

Focus on “big” ideas 

9 

10 

2 

5 

5 

0 

12 

5 

0 

3 

1 

52 

Embed problem solving 

7 

3 

5 

4 

8 

0 

11 

3 

1 

7 

3 

52 

Excite my students 

5 

6 

3 

2 

4 

0 

11 

2 

1 

7 

2 

43 

Develop students’ confidence 

6 

2 

0 

6 

5 

0 

12 

2 

0 

2 

2 

37 


Note: Teacher G logged 8 lessons and Teacher L logged 5 lessons. Responses from Teacher E, who logged only 1 lesson, are not included in this analysis. 


The logs capture a variety of practices 
within STEM classrooms and can, per¬ 
haps, speak to implementation of desired 
practices and/or the influence (or lack 
thereof) of specific professional develop¬ 
ment efforts in these classes. Perhaps the 
greatest caveat associated with teacher 
logs is that, like surveys, they represent 
self-reported data. Teachers may or may 
not take them seriously, they may “see” 
their lessons differently than an outsider 
might; and, regardless of the developer’s 
attention to ease-of-use, teachers may 
still view logs as an inconvenience and 
interruption in an already full day. 

Examining teaching practices under 
reforms targeted toward disadvantaged 
students is crucial. The schools employing 
the logs were not particularly advan¬ 
taged schools serving students with dem¬ 
onstrated interested and proficiency in 
STEM. About half of the students were 
in demographic groups underrepresent¬ 
ed in STEM and in college. Addition¬ 
ally, we found that the least experienced 
teachers responded to questions about 
the importance of lesson attributes some¬ 
what differently from more experienced 
teachers. Using this kind of teacher log 
can show whether new teachers are still 
learning about why certain attributes are 
important, and whether they need ad¬ 
ditional communication or professional 
development about these topics. 


Because the STEM schools in RTFs 
study were designed to be small, the 
mathematics and sciences departments 
comprised only three or four teachers, 
and we obtained logs from most of them. 
The teachers using the logs and other 
teachers in the school had similar lev¬ 
els of experience. In the school with the 
highest turnover and highest percentage 
of novice teachers, most of the teachers 
completing the logs were novices. Even 
though the characteristics of teachers re¬ 
sponding to the logs resemble those of 
all teachers in the school, we would not 
generalize these results to the school as 
the practices of these teachers may not 
represent those of teachers in other sub¬ 
jects. Rowan and Correnti (2009) cau¬ 
tion against generalizing results from a 
few teachers to the entire school because 
a few teachers may not represent school¬ 
wide practices. They suggest that data 
from 15-20 teachers are needed to yield 
school-level reliabilities. 

Responses to logs also gave us insights 
into the quality of responses, which we 
would not have had with a one-time sur¬ 
vey. Relatively high response rates and 
the variation in teacher responses give us 
confidence that almost all teachers did 
take the logs seriously and did not simply 
give the most desirable answer. Teachers 
understood that their responses would be 
confidential. Only aggregate results were 


shared with the school, and analytic re¬ 
ports listing individual teacher responses 
are de-identified and do not include the 
school name. If teachers had completed 
logs as part of a self-assessment, they 
might have felt less comfortable report¬ 
ing challenges, as they would have had 
an incentive to give the most desirable 
answer. Because they knew that the logs 
were going to be used to capture instruc¬ 
tional practices at the school, they may 
have felt more comfortable completing 
them. 

To facilitate teachers’ participation, we 
tried to make the logs easy for teachers 
to complete. Almost all of the questions 
were multiple choice, relying on teach¬ 
ers’ recollections of a given class period. 
Teachers were not burdened by hav¬ 
ing to look up additional information. 
Thus, logs should only have taken about 
10 minutes to complete. Having teach¬ 
ers complete logs over a specified time 
period and setting target dates for doing 
logs seemed to help teachers maintain a 
focus on this activity. Schools, districts, 
or researchers using logs might want to 
set specific goals for this activity, rather 
than having open-ended reporting. Face- 
to-face training and identifying a contact 
person who could help teachers with 
questions or challenges helped teachers 
become more comfortable with the logs. 
Schools, districts, or researchers using 
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logs might consider adopting these strat¬ 
egies. They could also offer teachers 
the option of completing a paper sur¬ 
vey instead of an on-line survey if that 
method seems better suited to them. In 
this study, teachers could complete paper 
surveys, but almost no one chose to do 
it that way. 

Teachers and administrators in the 
study suggested that the logs were a 
promising professional development tool 
that teachers could use on a regular basis 
to check the variability of instructional 
strategies they employed. They also com¬ 
mented on the potential for logs to be 
used within a school department as a 
collaborative tool within professional 
learning communities. The logs provid¬ 
ed teachers a common language around 
which to discuss improving classroom 
practices. Teachers can use log data 
to reflect on practice, determine areas 
of strength and challenge, and set goals 
for personal improvement. Administra¬ 
tors can use logs for many of the same 
purposes—to support teachers in exam¬ 
ining practice and setting goals. 
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